[core] Optimize partition discovery with projected manifest scans - #9321
[core] Optimize partition discovery with projected manifest scans#9321zhoulii wants to merge 2 commits into
Conversation
| } | ||
|
|
||
| /** Returns whether a manifest of this size is eligible for the configured cache. */ | ||
| public boolean isCacheable(long fileSize) { |
There was a problem hiding this comment.
Why not just use isCacheEnabled?
There was a problem hiding this comment.
isCacheable(fileSize) is intentional. When a manifest exceeds maxElementSize, ObjectsCache bypasses it even if caching is enabled. Using isCacheEnabled() would unnecessarily disable projected scans for
these large, non-cacheable manifests.
There was a problem hiding this comment.
Why we need to bypass it? Could you describe this scenario in more detail?
There was a problem hiding this comment.
Do you mean why cached reads cannot use projection as well?
Currently, projection is only supported by the file reader, while the cache stores and materializes full manifest entries. This change preserves the existing full-cache path for cacheable manifests and applies file-level projection only when the manifest cannot be cached.
Cache-side projected materialization could be added as a separate optimization. Is that what you are suggesting?
There was a problem hiding this comment.
When does the manifest size exceed the maxElementSize?
There was a problem hiding this comment.
With the default caching catalog settings, maxElementSize is 1 MB (cache.manifest.small-file-threshold), while manifest.target-file-size defaults to 8 MB. Therefore, normal manifest files can exceed maxElementSize.
| } else { | ||
| PartitionEntry.merge( | ||
| readManifest(m, PartitionEntry::fromManifestEntry, null, null), | ||
| readManifest( |
There was a problem hiding this comment.
Should just modify readManifest, Only readManifest knows whether direct access is currently possible, whether there are filtering conditions, and so on.
There was a problem hiding this comment.
Thanks, addressed. The projected/full-read selection is now centralized in an overloaded readManifest method, which checks cache eligibility and filters requiring complete manifest entries.
readPartitionEntries only supplies the projection, projected filter, converter, and aggregation consumer, while the projected path still aggregates entries in a streaming manner.
2 participants
Purpose
close #9320
Tests